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PREFACE 


Over  the  last  several  years,  a  new  approach  to  model-based  analysis 
has  been  developed  at  RAND.  This  approach — exploratory  analysis 
greatly  expands  on  traditional  analytic  approaches,  particularly 
sensitivity  analysis,  to  enhance  understanding  of  complex  problems, 
provide  a  wider  range  of  information  for  decisionmakers,  improve 
comparisons  among  alternative  modeling  venues,  and  thereby  enable 
greater  comprehension  of,  and  differences  among,  policy  options. 

This  documented  briefing  reviews  the  methodology  of  exploratory 
analysis  and  its  advantages  over  traditional  analysis  in  the  context  of  a 
search  for  the  preferred  weapon  mix.  We  find  that  exploratory  analysis 
can  enhance  decisionmaking  flexibility,  indicate  robustness  of  options, 
neutralize  risk,  and  provide  greater  understanding  of  the  policy 
problem  and  model. 

This  work  was  conducted  as  part  of  the  Theater  Modeling 
Improvement  Project  sponsored  by  the  Air  Force  Director  of  Modeling, 
Simulation,  and  Analysis  (AF/XOM).  The  Air  Force  point  of  contact 
for  the  study  is  Col  Ed  Crowder  of  the  Air  Force  Studies  and  Analysis 
Agency  (AFSAA).  A  companion  document.  Modeling  and  Simulation 
Infrastructure  to  Support  Adaptive  Planning  in  the  Air  Force,  DB-210-AF, 
describes  exploratory  analysis  more  generally  and  provides  brief 
examples  from  other  RAND  work.^ 

Project  AIR  FORCE 

Project  AIR  FORCE,  a  division  of  RAND,  is  the  Air  Force  federally 
funded  research  and  development  center  (FFRDC)  for  studies  and 
analyses.  It  provides  the  Air  Force  with  independent  analyses  of  policy 
alternatives  affecting  the  development,  employment,  combat  readiness, 
and  support  of  current  and  future  aerospace  forces. 


^  This  methodology  has  had  a  number  of  other  applications  to  date;  for  example,  see 
Steve  Bankes,  "Computational  Experiments  and  Exploratory  Modeling,"  CHANCE, 
Vol.  7,  No.  1, 1994,  pp.  50-57. 
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Research  is  performed  in  three  programs:  Strategy  and  Doctrine,  Force 
Modernization  and  Emplo3nnent,  and  Resource  Management  and 
System  Acquisition. 
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SUMMARY 


This  documented  briefing  discusses  the  advantages  of  exploratory 
analysis  over  more  traditional  model-based  analysis  in  the  context  of 
the  weapon  mix  problem.  We  illustrate  this  difference  first  by  example 
and  then  by  definition.  In  the  example,  we  walk  through  a  traditional 
analytic  approach  and  show  the  kinds  of  results  that  are  often 
observed.  We  then  perform  a  different  kind  of  analysis — ^based  on  a 
large  number  of  computational  experiments — on  the  same  problem, 
and  show  that  this  kind  of  analysis  provides  more  information  and 
keener  insights  than  we  originally  obtained.  We  continue  by  describing 
more  generally  this  methodology,  which  we  define  as  "exploratory 
analysis,"  and  demonstrate  its  benefits  to  the  decisionmaker  and  the 
analyst.  We  also  discuss  what  is  required  to  perform  this  type  of 
analysis. 

TRADITIONAL  ANALYSIS  VS.  EXPLORATORY  ANALYSIS: 

AN  APPLICATION 

We  illustrate  the  value  of  exploratory  analysis  by  applying  it  to  a 
problem  that  is  of  particular  relevance  to  the  Department  of  Defense 
and  the  services:  the  weapon  mix  problem  from  the  Deep  Attack/ 
Weapon  Mix  Study  (DAWMS).  A  best-estimate  scenario  is  developed 
and  a  similar  model  to  that  in  DAWMS  is  used  to  calculate  the 
preferred  weapon  mix.  We  then  perform  sensitivity  analysis  to 
measure  the  impact  of  uncertainty  in  the  weapon  reliability,  the  sortie 
rate,  and  the  deployment  schedule.  We  find  that  the  prescribed 
weapon  mixes  change  in  non-intuitive,  seemingly  erratic  ways  as  we 
marginally  change  these  inputs,  which  would  appear  to  imply  that 
either  the  model  is  faulty  or  that  there  are  errors  in  the  data.  Either 
way,  we  are  left  with  little  confidence  in  the  outcomes  of  the  study. 

When  we  extend  the  conventional  approach  by  applying  much  broader 
computational  experiments  than  those  used  originally,  we  discover  the 
explanation  for  the  seemingly  erroneous  results  obtained  earlier:  The 
model  and  data  are  not  causing  the  error;  rather,  the  culprit  is  the  way 
in  which  the  model  has  been  used.  Changing  and  expanding  the  use  of 
the  model  in  ways  carefully  explained  in  this  briefing  provide  a  clear 
explanation  of  the  previously  troubling  sensitivity  analysis  results. 


We  call  the  method  for  using  the  model  in  this  different  way 
"exploratory  analysis."  Broadly  stated,  exploratory  analysis  is  a 
methodology  designed  to  help  us  comprehend  complex  systems,  such 
as  (in  the  case  at  hand)  those  generally  represented  in  theater-level 
combat  models  with  many  imperfectly  known  parameters,  decisions, 
and  measures  of  effectiveness.  For  example,  we  begin  by  determining  a 
range  of  desirable  (or  tolerable)  outcomes  in  terms  of  such  measures  as 
time  to  complete  a  campaign  or  total  casualties.  We  then  explore  the 
spaces  of  scenario  conditions,  decision  options,  or  combinations  that 
correspond  to  this  range  of  outcome  values.  A  much  more 
comprehensive  set  of  trades  among  options  (than  that  gotten  from  the 
traditional  analysis)  can  then  be  presented  to  illuminate  the 
decisionmaking  process. 

In  the  case  at  hand,  exploratory  analysis  provides  the  means  for 
collecting  greater  information  about  the  weapon  mix  problem  and  the 
tool  we  are  using  to  evaluate  alternative  options.  Further,  we  are  able 
to  determine  the  full  range  of  weapon  mix  trades  that  accomplish 
campaign  objectives;  demonstrate  how  a  decisionmaker  can  see  the 
impact  of  imposing  additional  constraints  (such  as  on  cost  or  risk); 
suggest  how  a  weapon  mix  could  be  selected  that  is  robust  (or  as  robust 
as  possible)  across  contingencies;  and  reduce  or  neutralize  the  risk  of 
uncertainties  in  the  scenario.  As  this  document  shows,  without 
exploratory  analysis  in  the  case  examined  here,  the  decisionmaker 
would  have  been  left  with  unacceptable  results,  unnecessary  expenses 
to  fix  a  model  that  was  not  broken,  and  a  weapon  mix  study  that  still 
needed  to  be  done. 

EXPLORATORY  ANALYSIS  EXTENDS  TRADITIONAL  ANALYSIS 

To  understand  how  exploratory  analysis  extends  traditional  scenario- 
based  analysis,  we  now  step  back  and  more  generally  describe  how 
both  traditional  analysis  and  exploratory  analysis  are  performed. 

Traditional  Analysis 

Traditional  analysis  determines  the  value  or  impact  of  a  system  or 
policy  within  the  context  of  plausible,  best-estimate  current  or  future 
scenarios.  To  perform  traditional  analysis  (as  stated  in  the  description 
of  the  present  case  study),  we  first  define  these  scenarios.  Concerns 
about  uncertain  or  unknown  scenario  conditions  and  specific  data 
items  often  stimulate  large-scale  efforts  to  determine  precisely  the  best 
(or  most  acceptable)  estimates  of  the  data  for  the  analysis  as  a  whole 
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and,  in  particular,  the  model(s)  used  to  calculate  performance 
measures.  Sensitivity  analysis  is  used  to  augment  the  best-estimate 
analysis  to  determine  the  impact  of  variations  or  errors  in  these 
estimates. 

Frequently,  the  sensitivity  analysis  produces  troubling  results: 
changing  the  input  conditions  causes  some  output  values  to  change 
erratically — and  not  infrequently  in  non-intuitive  ways.  Analysts  and 
decisionmakers  often  interpret  these  outcomes  as  errors  in  the  model, 
which  sometimes  leads  to  the  model  (and  the  analysis)  being 
discredited.  One  recourse  is  to  instigate  a  major  (and  most  likely  very 
costly)  review  of  the  model  and/or  the  associated  databases  in  an 
attempt  to  fix  the  errors  or  to  increase  "realism"  by  including  a  broader 
scope  or  more  details.  These  efforts  may  make  the  model  only  more 
cumbersome  and  difficult  to  validate.  It  is  often  not  possible  to  know 
what  value  has  actually  been  added  to  the  decisionmaking  process. 

Exploratory  Analysis 

Exploratory  analysis  differs  from  traditional  analysis.  Operationally,  it 
changes  the  way  the  model  is  used:  the  model  is  run  many  times  with 
many  different  input  levels  as  opposed  to  one  best-estimate  case  run 
followed  by  a  (probably)  fairly  limited  sensitivity  analysis. 
Methodologically,  the  difference  is  starker:  exploratory  analysis 
represents  a  fundamentally  different  way  of  looking  at  the  problem. 
While  traditional  analysis  works  in  a  sense  from  the  inside  out — the 
solution  is  found  and  then  the  area  around  it  examined — exploratory 
analysis  features  an  "outside  in"  approach:  A  large  set  of  plausible 
scenario  conditions,  decision  options,  and  desirable  outcomes  is 
examined,  and  then  a  preferred  solution  is  selected. 

As  this  document  will  make  explicit,  this  methodology  has  a  number  of 
advantages  over  the  traditional  approach. 

-  Exploratory  analysis  can  provide  greater  insights  than  can 
traditional  methods,  including  the  information  that  explains  and 
sometimes  resolves  troubling  sensitivity  analysis  results — ^without 
resorting  to  changes  in  the  model  or  data. 

-  Exploratory  analysis  can  help  decisionmakers  choose  options  that 
are  robust  across  different  scenario  conditions,  operational  or 
technical  preferences,  and  costs. 

-  Exploratory  analysis  can  be  an  effective  tool  in  the  process  of  the 
verification  and  validation  of  the  models  used. 
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As  systemic  complexity  is  a  feature  central  to  many  problems, 
exploratory  analysis  can  be  and  has  been  useful  in  many  contexts.  Our 
case  study  makes  its  use  clear  for  military  modeling  and  simulation;  it 
has  also  been  used  to  address  problems  in  areas  as  divergent  as  climate 
change,  information  infrastructure,  and  investment  strategy.  As  we 
will  show,  however,  there  are  limits  on  the  scale  of  explorations  we  can 
undertake  in  different  problems.  To  perform  the  computational 
experiments  that  are  essential  for  undertaking  this  analysis  (often 
consisting  of  tens  or  hundreds  of  thousands  of  computer  runs), 
considerable  computational  horsepower  must  be  made  available  to  a 
study  team.  In  addition,  the  means  for  defining,  implementing,  and 
managing  the  experiments  as  well  as  the  vehicle  for  characterizing  and 
analyzing  the  outputs  must  be  provided.  In  other  words,  the  use  of 
exploratory  analysis  is  often  bound  by  infrastructural  constraints. 

In  this  documented  briefing,  we  suggest  ways  to  enhance  the 
infrastructure  for  exploratory  analysis  and  thus  make  the  approach  as 
practical  as  possible.  We  believe  that  the  applicability  of  exploratory 
analysis  is  very  wide  and  will  provide  marked  improvements  in  how 
analysis  can  aid  the  decisionmaking  process. 
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This  briefing  presents  an  innovative  approach  to  decisionmaking  called 
exploratory  analysis;  we  explain  it  in  the  context  of  a  problem  that  is  of 
particular  importance  to  the  Department  of  Defense  and  services. 
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The  problem  is  taken  from  the  Deep  Attack/Weapon  Mix  Study 
(DAWMS),  which  entails  a  determination  of  the  appropriate  mix  of 
weapons  and  forces  for  performing  the  deep  attack  mission.  Specifically, 
we  will  look  at  two  questions  that  are  part  of  this  study.  First,  what 
weapons  are  needed,  given  the  set  of  scenarios  judged  most  probable? 
Second,  what  would  be  the  impact  of  changing  the  availability  of  these 
weapons? 

While  we  look  at  an  actual  problem  here,  the  purpose  is  not  to  recommend 
specific  acquisition  decisions.  Rather,  we  are  interested  in  comparing 
exploratory  analysis  to  traditional  analysis  and  demonstrating  how  this 
methodology  could  be  applied  in  reaching  such  recommendations. 
Therefore,  we  have  taken  the  liberty  of  using  a  model  and  a  database  that 
were  readily  available  to  us  at  the  beginning  of  this  study  and,  although 
similar,  were  not  specifically  used  in  DAWMS. 
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Conclusions 


Traditional  analysis  provides  useful  but  limited 
information  for  a  decisionmaker. 

Improving  the  model  or  data  alone  can  be 
inefficient  and  may  not  solve  the  problem. 

Advantages  of  exploratory  analysis 

-  Decisionmaking  flexibility 

-  Robustness  across  contingencies 

-  Risk  neutralization 

-  Greater  understanding  of  the  model 
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We  will  reach  three  basic  conclusions  here.  First,  traditional  analysis  of 
the  weapon  mix  problem  is  useful  but  incomplete  in  providing 
information  to  the  decisionmaker.  We  are  not  saying  that  traditional 
analysis  is  bad,  but  rather  that  it  needs  to  be  broadened. 

Second,  we  will  see  that,  on  encountering  counterintuitive  results  in  this 
analysis,  the  standard  remedies  of  seeking  resolution  within  the  model 
or  via  data  modifications  is  simply  wrong.  The  real  problem  is  how  the 
model  has  been  used  or,  more  properly,  how  it  has  not  been  used  to  its 
fullest  extent.  Further,  if  fixing  the  model  or  data  has  any  ameliorating 
effect  at  all,  it  may  not  be  the  most  efficient  way  to  achieve  better  results. 

Finally,  we  will  find  that  exploratory  analysis  has  four  advantages  over 
traditional  analysis.  First,  this  methodology  increases  flexibility  for 
decisionmakers,  including  planners,  operators,  and  budgeters  (as  we 
will  demonstrate).  Second,  such  modeling  allows  the  assessment  of  the 
robustness  of  a  particular  weapon  mix  across  different  contingencies, 
including  major  regional  conflicts,  lesser  regional  conflicts,  and  missions 
other  than  war.  Third,  it  allows  us  effectively  to  neutralize  risk  that 
comes  from  imperfect  information,  as  well  as  an  uncertain  future. 

And  fourth,  we  will  see  that  this  methodology  affords  us  greater  insights 
into  the  policy  problem  as  well  as  a  better  understanding  of  the  model. 
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Outline  | 

•  A  traditional  analysis  with  some 
conclusions 

•  What  is  really  happening:  an 
introduction  to  exploratory  analysis 

•  The  methodology  of  exploratory 
analysis 

•  How  can  exploratory  analysis  help 
decisionmakers? 

•  Uses  and  requirements  of  exploratory 
analysis 
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We  will  compare  exploratory  analysis  to  traditional  analysis  by  first 
demonstrating  a  traditional  analysis  of  the  weapon  mix  problem.  This 
will  lead  to  some  troubling  results.  We  will  then  explain  what  is  really 
happening  by  applying  exploratory  analysis.  In  the  third  section,  the 
methodology  is  defined  and  described  in  greater  generality.  In  the 
fourth  section,  a  variety  of  examples  are  given  to  help  decisionmakers 
see  what  benefits  exploratory  analysis  may  provide.  Finally,  we 
discuss  the  potential  and  actual  uses  of  exploratory  analysis,  as  well  as 
what  is  required  for  its  routine  application. 

We  now  proceed  with  the  first  part,  a  traditional  scenario-based 
analysis  of  the  weapon  mix  problem,  followed  by  some  general 
conclusions  from  this  analysis. 
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The  Analytic  Plan 


•  Select  a  scenario 

-  Location 
-Time 

-  Systems:  targets,  threats,  aircraft,  and  weapons 
-Constraints:  attrition,  CONOPs,  budget 

-  MOEs:  targets  destroyed,  time  to  complete  phases 

•  Determine  the  best  weapon  mix  based  on  most  iikeiy 
conditions 

•  Conduct  sensitivity  anaiysis 
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The  first  step  in  a  traditional  analysis  is  the  selection  of  a  scenario. 

Much  of  the  work  to  determine  the  best  weapon  mix  occurs  in  defining 
the  scenarios.  This  involves  the  location  and  time  of  the  operation,  the 
systems  under  consideration  (including  the  targets,  threats,  platforms, 
and  weapons),  the  necessary  constraints  on  the  problem  (such  as 
attrition,  concepts  of  operation  (CONOPs),  or  budget),  and  measures  of 
effectiveness  (MOEs),  such  as  the  value  of  the  targets  destroyed  or  the 
time  spent  in  completing  the  phases  of  the  campaign.  An  analysis  of  the 
scenario  ensues,  which  requires  an  estimation  of  the  most  likely 
conditions  for  the  scenario — the  best  estimates  of  the  values  of  the 
parameters  in  the  model.  As  a  second  step  in  this  process,  the  best 
weapon  mix  is  determined,  often  calculated  using  a  model,  from  the 
best-estimate  scenario.  As  a  result  of  examining  the  model  output, 
changes  to  the  base  case  may  be  made  to  correct  for  unforeseen 
outcomes. 

The  last  step  in  the  traditional  analysis  is  to  conduct  sensitivity  analysis. 
Sensitivity  analysis  examines  variations  to  input  values  from  the  best- 
estimate  (base)  case.  Its  purpose  is  to  gauge  how  sensitive  outcomes  are 
to  small  changes  in  the  parameters,  and  in  this  way  address  the 
potential  effects  of  uncertainty. 
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CONSTRAINTS 


t:- . . - .  .  . - - - 

y  ATTRITION  1  CONORS 

f  BUDGET  p 

Weapon  Mix 
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The  analytic  process  described  in  the  preceding  chart  is  often 
undertaken  with  the  aid  of  computer  models.  In  our  analysis,  we,  too, 
rely  on  a  model  to  determine  the  best  weapon  mix.  Our  model  process 
employs  an  optimizing  model  and  is  diagnosed  above.^ 

First,  the  measure  of  effectiveness  is  designated  to  drive  the 
optimization.  We  may  select  from  a  variety  of  measures  to  drive  the 
optimization,  such  as  time  to  complete  campaign  phases,  while  we  still 
capture  and  calculate  other  measures,  such  as  targets  destroyed  or  cost. 
Then,  we  input  the  systems  (the  quantity,  quality,  and  characteristics  of 
the  aircraft,  weapons,  and  targets)  and  constrain  the  model 
appropriately  (limiting  attrition  or  budget,  or  the  manner  in  which  the 
systems  must  perform).  Next,  we  run  the  optimizer,^  which  produces 
three  pieces  of  information.  The  first  is  the  best  possible  value  of  our 
objective;  the  value  of  other  measures  of  effectiveness  can  also  be 


^  The  reader  might  question  whether  optimization  is  in  fact  the  right  tactic,  given  the 
imcertainty  inherent  in  warfighting.  Tlie  point  is  well-taken,  but  we  proceed  for  two 
reasons.  First,  we  are  demonstrating  the  exploratory  analysis  approach  in  a  given 
context;  that  given  context  assumes  that  requirements  have  been  specified  and  an 
optimization  will  be  performed.  Second,  the  approach  we  discuss  will  work  just  as 
well  (and  in  some  respects,  even  better)  with  other  types  of  models,  such  as 
simulations. 

^  The  actual  model  used  here  was  the  Conventional  Targeting  Effectiveness 
Model  (CTEM).  We  note  again  that  DAWMS  used  a  different  optimization  model; 
but  any  model,  either  prescriptive  or  descriptive,  could  be  used. 


6 


calculated.  Second,  an  allocation  is  given;  this  is  a  schedule  indicating 
what  aircraft /weapon  combination  attacks  each  target,  how  it  is 
employed,  and  in  what  sequence.  Third,  we  calculate  the  mix  of 
weapons  to  accomplish  the  task.  The  weapon  mix  is  probably  the  most 
important  piece  of  information  to  us,  as  it  represents  the 
recommendation  to  the  decisionmaker:  it  is  the  combination  of 
weapons  we  need  to  attain  the  optimal  value  of  our  measure  of 
effectiveness. 
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Sample  Scenario 


SWA,  2001 

OBJECTIVE:  Minimize  time  to  compiete  campaign 

TARGETS 

DIA  Outyear  Threat  Assessment 

Goals  on  Targets:  CINC  objectives  from  XOFW 

WEAPONS 

Inventory  from  XOFW  projections 
Munition  effectiveness  -  SABSEL 

PLATFORMS 

Deployment  -  Nimble  Dancer  TPFD  Schedule 
Sortie  Rates  -  WMP  Data 

Special  Features 

no  attrition,  good  weather 
no  TEL  hunting,  holdout  for  SEAD 

O  10/96  RAND 


We  now  proceed  to  demonstrate  the  analytic  and  modeling  process  we 
have  just  described.  In  phase  one,  we  determine  the  best  estimate  for 
all  the  inputs  that  define  the  scenario. 

Our  results  are  based  on  a  sample:  SWA  2001  scenario.^  The  measure 
of  effectiveness  selected  as  the  objective  is  the  time  to  complete  the 
entire  campaign  (which  we  will  minimize).  Data  are  taken  from  a 
variety  of  authoritative  sources,  as  noted  on  the  next  slides. 

To  simplify  our  examination  of  this  scenario,  we  have  assumed  some 
special  features  regarding  attrition  and  weather  that  are  particularly 
optimistic.  However,  this  should  not  pose  a  problem,  since  our 
purpose,  as  we  have  already  mentioned,  is  simply  to  demonstrate  the 
difference  between  traditional  and  exploratory  analysis. 


^This  scenario  was  assembled  by  Keith  Henry  at  RAND.  We  note  that  this  is  not  one 
of  the  DAWMS  scenarios. 


Best-Estimate  Results  | 

OBJECTIVE 

1 

CAMPAIGN  IS  COMPLETED  IN  22  DAYS  I 

DECISIONS 

TOTAL  Weapon  1  USED: 

15,502 

TOTAL  Weapon  2  USED: 

6,960 

TOTAL  Weapon  3  USED: 

5,003 

g  loss  RAND 

Once  the  scenario  is  completely  defined,  we  move  on  to  step  two  in  the 
analytic  process.  Running  the  model  with  all  the  input  information 
described  produces  the  results  shown  here:  The  minimal  campaign 
completion  time  is  22  days.  To  achieve  this  best-estimate  result,  the 
model  tells  us  to  inform  the  decisionmaker  that  15,502  of  Weapon  1 
should  be  in  inventory,  along  with  6,960  of  Weapon  2  and  5,003  of 
Weapon  3  (among  other  munitions).'* 

A  striking  observation  is  the  recommendation  to  acquire  so  many  of 
Weapon  1.  Given  this,  one  wonders  what  would  happen  if  Weapon  1 
were  not  to  work  as  advertised — suppose  jamming  or  some  other 
impediment  to  Weapon  I's  effectiveness  were  to  occur.  Would  this 
have  a  detrimental  impact  on  the  ability  to  complete  the  campaign  in  a 
timely  fashion?  These  and  other  questions  lead  us  into  step  three  of  the 
traditional  analysis:  sensitivity  analysis. 


*The  model  determines  the  quantities  for  each  weapon  type.  We  have  aggregated 
these  into  the  three  weapon  classes  (Weapon  1,  Weapon  2,  and  Weapon  3)  as  a  matter 
of  convenience. 
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To  undertake  sensitivity  analysis,  we  need  first  to  decide  on  the 
parameters  to  vary.  The  criterion  for  choosing  a  particular  parameter 
generally  is  an  a  priori  belief  that  a  change  in  it  will  result  in  some 
change  in  our  measure  of  effectiveness. 

In  the  previous  chart,  we  saw  that  the  prescribed  level  of  Weapon  1  was 
quite  high  relative  to  Weapon  2  and  Weapon  3.  Therefore,  we  would 
likely  expect  a  change  in  the  effectiveness  of  these  weapons  to  have  a 
disproportionately  great  effect  on  our  ability  to  complete  the  campaign 
in  22  days;  hence,  the  first  parameter  we  choose  to  vary  is  Weapon  1 
reliability.  We  scale  back  this  reliability  to  see  how  it  affects  the 
number  of  days  to  complete  the  campaign. 

The  other  parameters  we  examine  affect  the  number  of  weapons  that 
can  be  delivered  on  targets  within  a  set  amount  of  time:  the  sortie  rate 
and  the  time-phased  force  deployment  (TPFD)  schedule.  As  a  first 
attempt  to  understand  the  effect  of  sortie  rate  changes,  we  will  simply 
change  the  daily  rate  across  the  board  without  any  preferential 
treatment  of  any  particular  aircraft.  Changes  in  the  deployment 
schedule  are  also  made  without  preference  for  any  platform  type. 

Here,  we  degrade  the  entire  schedule:  The  graph  above  on  the  left 
shows  the  baseline  TPFD  schedule;  the  graph  on  the  right  reflects 
platforms  arriving  at  the  theater  of  operations  on  the  same  schedule, 
but  with  20  percent  fewer  arriving  on  each  day.  Note  that  the  schedule 
is  cumulative — ^by  the  24th  day,  all  the  platforms  that  are  coming  have 
arrived.  It  is  also  monotonic,  because  the  model  has  no  attrition. 
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In  sum,  in  this  initial  sensitivity  analysis  we  measure  the  variance  of  the 
parameters  as  a  simple  degradation  from  their  baseline  (best-estimate) 
levels. 
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The  results  of  this  sensitivity  analysis  can  be  viewed  graphically.  It  is 
easiest  to  visualize  the  results  by  holding  one  of  the  parameters  constant 
and  by  representing  changes  in  it  with  a  series  of  graphs.  Thus,  we  hold 
sorties  constant  at  100  percent  (we  have  left  sorties  at  their  baseline 
level).  The  above  chart  shows  the  effect  on  the  objective  (days  to 
complete  the  campaign)  from  varying  the  Weapon  1  reliability  and  the 
deployment  schedule.  The  original,  best-estimate  solution  value — 22 
days  to  complete — can  be  found  in  the  front  corner,  where  the  two 
parameters  are  at  their  best-estimate  (100  percent)  values. 

The  graph  is  monotonically  increasing  over  its  entire  range — as  either 
parameter  is  degraded  independently,  the  value  of  the  objective  always 
increases,  and  takes  longer  to  complete  the  campaign.  This  makes 
intuitive  sense,  as  might  the  fact  that  degrading  the  deployment 
schedule  seems  to  have  relatively  greater  impact  than  does  poor 
Weapon  1  reliability  (since  we  can  substitute  between  weapons  when 
Weapon  1  weapons  are  less  effective).  It  can  be  shown  that  degrading 
the  sortie  rate  has  a  similarly  monotonically  increasing  relationship  with 
the  objective. 

The  slope  of  the  graph  is  also  quite  gradual.  All  of  this  is  good  news 
from  the  point  of  view  of  traditional  sensitivity  analysis:  The  gentle 
grade  means  that  a  small  error  in  measuring  the  parameters  will  result 
in  a  small  error  in  the  value  of  the  objective;  the  monotonicity  means 
that  this  error  will  also  be  predictable  (we  know  that  estimating  a 
parameter  level  too  optimistically  will  always  result  in  an  increase  in  the 
days  to  complete,  for  example). 
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The  results  on  the  previous  chart  would  represent  a  rather  thorough 
sensitivity  analysis  by  the  standards  of  many  studies.  We  might  go 
even  further,  though,  and  question  whether  the  apparent  smoothness 
of  the  response  surface  is  an  artifact  of  the  coarseness  with  which  we 
have  sampled  from  the  parameter  space.  That  is,  some  roughness  in 
the  graph  might  be  masked  if  it  happens  to  fall  between  the  tick  marks 
designated.  For  this  reason,  we  undertake  a  more  detailed  sensitivity 
analysis. 

In  this  chart,  sorties  are  degraded  to  60  percent,  and  we  look  at  deploy¬ 
ment  between  60  and  90  percent  (drawn  down  in  5-percent  incre¬ 
ments)  and  Weapon  1  reliability  between  100  and  50  percent  (also  in 
5-percent  increments).  Note  that  the  60  percent  sortie  level  has  raised 
the  surface  (it  takes  longer  to  complete  the  campaign).  This  graph 
reinforces  the  good  news  of  the  last:  the  results  are  monotonic  and 
gently  sloping.®  A  small  parameter  misestimate  will  lead  to  a  small, 
predictable  error  in  the  measure  of  effectiveness. 

Unfortunately,  these  intuitive  observations  are  not  the  end  of  the  story. 
Recall  that  the  recommendation  we  wish  to  give  the  decisionmaker  is 
not  the  number  of  days  to  complete  the  campaign,  but  rather  how 


^Although  the  chart  may  appear  to  be  a  little  "bumpy,"  we  can  assure  the  reader  that 
it  is  really  monotonic.  The  apparent  roughness  occurs  because  of  the  perspective¬ 
drawing  limitations  of  the  drawing  software. 
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many  of  each  type  of  munition  to  have  in  inventory.  Hence,  we  should 
be  concerned  with  not  only  the  sensitivity  of  the  objective,  but  also  of 
the  decisions:  how  many  weapons  do  we  need  if  conditions  change? 
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This  chart  looks  at  the  sensitivity  of  the  Weapon  1  decision,  within  the 
same  parameter  values  as  those  in  the  previous  chart.  The  axes  are 
reversed  for  reasons  of  perspective. 

Clearly,  what  we  observe  is  quite  different  from  what  we  saw  in  the  last 
chart:  There  is  considerable  nonmonotonicity  in  this  graph.  These 
features  are  apparent  with  respect  to  each  of  the  parameters 
individually  (including  sorties,  as  can  be  shown). 

What  does  this  mean?  Consider  the  case  in  which  the  deployment 
schedule  is  held  at  80  percent  and  Weapon  1  reliability  is  decreased. 
Initially,  the  number  of  Weapon  1  used  by  the  model  decreases,  as 
makes  intuitive  sense.  However,  we  can  see  that  a  continued  degrada¬ 
tion  in  reliability  leads  to  more  of  Weapon  1  being  used.  This  is  an  odd 
result:  to  have  less,  then  more,  then  less,  then  more  of  Weapon  1  used 
as  Weapon  1  reliability  degrades.  What  is  more  troubling  is  that  these 
same  sorts  of  nonmonotonic  changes  in  Weapon  1  occur  as  the 
deployment  is  degraded. 

One  explanation  for  this  behavior  might  be  that  the  number  of 
Weapon  1  increases  to  compensate  for  decreases  in  their  reliability  up  to 
a  certain  point.  Once  Weapon  1  reliability  falls  beyond  a  certain  point, 
our  preferred  solution  is  to  use  more  of  some  other  kind  of  weapon 
and,  hence,  fewer  of  Weapon  1.  However,  we  would  not  expect  this 
behavior  to  repeat  itself  as  Weapon  1  reliability  declines  or  to  see  it 
across  different  levels  of  deployment,  which  is  the  case.  Thus,  we  are 
left  to  question  what  is  really  going  on  in  this  analysis. 
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Not  only  are  the  model  results  troubling,  but  it  is  impossible  to  predict 
whether  more  or  fewer  of  Weapon  1  are  really  required.  A  small 
change  or  misestimate  in  these  uncertain  parameter  values  (Weapon  1 
reliability  and  deployment  schedule)  might  cause  a  weapon  usage  that 
is  substantially  different  from  that  found  to  solve  the  best-estimate 
scenario. 
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Conclusions  from  Sensitivity  Analysis 


Model  algorithms  are  unstable. 

Input  data  must  be  extremely  precise. 

Sufficient  munitions  should  be  acquired  to  cover 
uncertainties. 
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What  conclusions  might  we  reach  based  on  this  sensitivity  analysis? 

First,  we  might  conclude  that  there  is  a  grave  problem  with  the  model; 
perhaps  the  algorithms  are  unstable,  or  maybe  we  have  some  numerical 
problem.  A  linear  programming  expert  might  conclude  that  we  are  simply 
alternating  between  comer-point  solutions.  To  address  this  problem,  we 
would  likely  seek  out  expertise  in  the  model's  code.  Perhaps  we  would 
even  recommend  putting  greater  detail  or  better  representations  in  the 
model. 

Second,  we  might  look  at  the  measurement  of  the  inputs:  If  we  can  ensure 
that  the  parameters  are  all  accurate  and  precise  going  into  the  calculation, 
we  have  no  need  for  the  sensitivity  analysis  because  the  best  estimates  are 
absolutely  correct  and  we  assert  there  will  be  no  deviation  from  them. 

Then,  the  ruggedness  that  occurs  around  the  best  estimate  weapon 
combination  is  no  longer  of  consequence.  To  improve  these  measurements, 
we  might  (for  instance)  assemble  committees  or  battle  staffs  to  derive 
highly  precise  inputs.  But,  in  making  this  assertion,  we  surely  are  only 
denying  the  enormous  uncertainties  of  predicting  future  scenarios, 
battlefield  outcomes,  and  system  operations. 

Either  of  these  remedies  would  take  some  time  to  complete  and  may  lead 
to  a  large  capital  expense.  Thus,  a  third  conclusion  (which  is  more  near 
term)  might  be  that  we  should  "buy  the  problem  out":  acquire  enough 
munitions  to  cover  all  the  uncertainty.  After  all,  perhaps  this  is  just  the 
"noise"  caused  by  the  large  uncertainties  in  this  problem,  and  the  only 
thing  we  can  do  is  either  suffer  the  risk  or  buy  it  out.  The  problem  with 
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this  remedy,  naturally,  is  simply  one  of  feasibility.  There  are  likely 
many  situations  in  which  buying  out  the  largest  deviation  is  fiscally 
infeasible. 

At  this  juncture,  we  have  completed  the  traditional  analysis,  although  it 
has  left  us  with  some  generally  unsatisfying  conclusions.  The  alterna¬ 
tives  all  seem  costly.  Analysts  and  decisionmakers  look  at  our 
troubling  conclusions  suspiciously.  Not  only  is  it  likely  that  the  model 
and  the  analysis  have  been  discredited,  but  we  now  face  the  dilemma  of 
spending  additional  time  and  money  to  fix  the  model  or  data,  or  be  left 
with  a  weapon  mix  study  that  still  needs  to  be  done. 
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We  now  move  on  to  get  to  the  bottom  of  the  troubling  results  that 
plagued  our  traditional  analysis. 
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These  Conclusions  Are  Incorrect 


The  model  is  not  the  problem. 

The  data  are  not  the  problem. 

It  is  not  necessary  to  acquire  additional  munitions. 

Why  not? 


20  RAND 

As  familiar  as  the  preceding  conclusions  might  seem,  they  are,  at  least 
in  this  case,  not  correct:  Neither  the  model  nor  the  data  are  the 
problem,  and  acquiring  additional  munitions  is  not  necessary.  As  we 
shall  show,  the  real  problem  is  how  the  model  has  been  used,  not  the 
model  itself. 
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Optimal  Solutions 

Weaoon  Mix 

Weaoon  1  reliabilitv 

Weapon  1 
Weapon  2 

12,781 

90%  85%  80% 

9,570 

22  Days  j 

Weapon  3 

5,794 

I 

Weapon  1 

12,927 

I 

Weapon  2 

9,617 

22  Days 

Weapon  3 

5,206 

Weapon  1 

12,446 

Weapon  2 

10,291 

23  Days 

Weaoon  3 

6,101 
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To  justify  our  assertion,  consider  the  following  example.  This  chart 
shows  three  levels  of  Weapon  1  reliability  across  the  upper  right.  On 
the  left  are  the  optimal  weapon  mixes  for  each  of  these  cases,  with  the 
corresponding  number  of  days  to  complete  the  campaign  shown  in  the 
box. 

When  Weapon  1  reliability  is  90  percent,  the  campaign  finishes  in 
22  days;  and  we  are  told  to  use  12,781  of  Weapon  1, 9,570  of  Weapon  2, 
and  5,794  of  Weapon  3  (the  top  weapon  mix).  Similarly,  at  85-percent 
Weapon  1  reliability,  we  finish  in  22  days  and  are  told  to  use  the  middle 
weapon  mix;  at  80  percent,  we  finish  in  23  days  and  use  the  bottom 
mix. 

These  different  weapon  mixes  exhibit  precisely  the  fluctuations  seen 
before.  As  Weapon  1  reliability  falls  from  90  to  80  percent,  for  example, 
the  number  of  Weapon  1  required  first  increases,  and  then  decreases, 
while  the  number  of  Weapon  3  first  decreases  and  then  increases. 

Although  these  weapon  mixes  differ  from  one  another,  what  would 
happen  to  the  value  of  our  objective  (the  number  of  days  to  complete 
the  campaign)  in  the  90-percent  case  if  we  were  to  use  the  weapon 
mixes  calculated  for  the  85-percent  or  80-percent  cases?  Since  we  know 
that  22  days  is  an  optimal  value,  we  won't  do  any  better  than  this,  but 
how  much  worse  will  we  do  with  the  alternative  mixes? 
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As  it  happens,  we  will  do  no  worse  at  all,  technically:  in  both  cases,  we 
can  still  finish  the  campaign  in  about  22  days.  While  in  truth  the  first 
weapon  mix  performs  fractionally  better  than  the  others  (the  linear 
program  does  find  the  best  fractional  solution),  when  rounded  to  the 
nearest  day  there  is  no  difference.  These  fractional  savings  are  well 
within  the  overall  calculation  and  likely  to  be  irrelevant  to  the  decision¬ 
maker  when  compared  with  accompanying  differences  in  other 
measures  of  interest,  such  as  cost  or  flexibility.  Hence,  we  have  found, 
at  least  in  terms  of  days  to  complete  the  campaign,  equally  good,  near- 
optimal  alternative  weapon  mixes. 

Similarly,  what  would  happen  to  our  objective  value  if  we  were  to  take 
the  weapon  mixes  calculated  for  90-  and  80-percent  Weapon  1  reliability 
and  use  them  in  the  model,  while  the  Weapon  1  reliability  is  set  to 
85  percent? 
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More  Solutions 

Weapon  Mix 

Weapon  1  reliability 

1 

Weapon  1 

12,781 

85%  80% 

Weapon  2 

9,570 

22  Days 

22  Days 

1 

Weapon  3 

5,794 

Weapon  1 

12,927 

Weapon  2 

9,617 

22  Days 

22  Days 

Weapon  3 

5,206 

Weapon  1 

12,446 

)•* 

Weapon  2 

10,291 

22  Days 

22  Days  23  Days 

Weapon  3 

6,101 
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Again,  the  value  of  our  objective  (22  days)  remains  unchanged  for  these 
alternative  weapon  mixes.  Completing  the  example,  we  plug  in  the 
mixes  from  90-  and  85-percent  reliability  and  run  our  model  at  80 
percent. 
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There  Are  Many  Ways  to 

Accomplish  the  Mission 

Weapon  Mix 

Weapon  1  reliability 

Weapon  1 

Weapon  2 

12,781 

9,570 

90% 

85% 

80% 

22  Days 

22  Days 

23  Days  ! 

Weapon  3 

5,794 

Weapon  1 

12,927 

Weapon  2 

9,617 

22  Days 

22  Days 

23  Days 

Weapon  3 

5,206 

Weapon  1 

12,446 

i 

1 

Weapon  2 

10,291 

22  Days 

22  Days 

23  Days 

Weaoon  3 

6,101 

24 

io«£  RAND 

As  by  now  we  would  expect,  the  objective  value  is  unchanged  at 
23  days  to  complete  the  campaign.  So  what  is  going  on  here? 

What  this  example  tells  us  is  that  the  solutions  calculated  by  the 
optimization  model  are  not  unique;  different  combinations  of  weapons 
can  be  used  with  essentially  the  same  impact  on  the  number  of  days  to 
complete  the  mission.  And  the  few  solutions  found  here  are  probably 
not  coincidental:  We  are  led  to  suspect  that  there  are  likely  many 
combinations  that  will  do  as  good  a  job.  This  supports  our  intuition 
that  there  are  a  variety  of  ways  to  successfully  fight  a  war.  This  is 
encouraging  news,  but  can  we  find  all  of  them?  Yes,  by  systematically 
searching  across  the  space  of  weapon  mixes.  To  do  this,  we  no  longer 
use  the  model  in  the  typical  way  to  find  an  optimal  solution.  Instead, 
we  are  changing  it  into  a  search  engine  to  uncover  all  the  optimal,  or 
near-optimal,  solutions. 
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Alternative  Weapon  Mixes: 

22-23  Days  to  Complete  Campaign 


Weapon  1  rel.=100% 
Sorties=100% 
Deployment=1 00% 


Weapon  1 


9,000 

7,000  ' 

5,000*’ 


13,000 
9,000  Weapon  2 
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This  chart  presents  the  findings  of  such  a  search.  Discrete  sampling 
found  196  different  combinations  of  Weapons  1, 2,  and  3  that  all 
produced  a  value  of  22-23  days  to  complete  the  campaign.  Each  combi¬ 
nation  of  weapons  (rounded  up  to  the  nearest  hundred)  is  represented 
by  the  position  and  height  of  each  column  (the  point  corresponding  to  a 
particular  combination  is  at  the  top  of  each  column).  Note  that  the 
original  solution  is  found  among  these  solutions  (it  would  be  troubling 
if  it  were  not,  of  course). 

In  actuality,  any  weapon  mix  that  lies  within  the  boundary  formed  by 
the  top  of  these  bars  is  a  solution.  Instead  of  knowing  just  the  one 
solution  picked  up  by  traditional  analysis,  we  can  calculate  a  wide  range 
of  alternative  weapon  mixes,  all  of  which  are  clustered  in  a  particular 
region — a  strip  stretching  from  the  area  of  low  Weapon  2  levels  and  high 
Weapon  1  levels  to  the  area  of  low  Weapon  1  and  high  Weapon  2. 

This  chart  has  two  salient  features  worth  mentioning,  both  of  which 
contain  important  information  for  the  decisionmaker  and  neither  of 
which  is  apparent  via  traditional  analysis.  First,  this  graph  indicates  the 
way  to  trade  off  weapons  to  accomplish  the  mission  in  22-23  days. 
Specifically,  note  that,  at  the  corner  in  which  Weapon  1  levels  are  low 
but  Weapon  2  levels  are  high,  we  need  to  use  more  of  Weapon  3  than  in 
the  opposite  case.  Why  might  this  be?  The  answer  involves  the  nature 
of  the  platforms  being  used.  While  fighters  carry  all  three  weapon 
t5^es,  bombers  carry  only  Weapon  1  and  Weapon  3.  Therefore,  at  low 
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numbers  of  Weapon  1,  the  bombers  use  an  increasingly  large  number  of 
Weapon  3  as  their  only  alternative  weapon  type.  This  will  cause  a 
disproportionate  substitution  into  Weapon  3  at  this  comer  and  create 
the  observed  skewness. 

Second,  we  may  ask  why  there  are  no  weapon  mixes  in  the  front  or 
back  portions  of  the  box.  After  thinking  about  the  region  in  front,  we 
conclude  that  the  small  numbers  of  Weapons  1  and  2  do  not  allow  us 
successfully  to  attack  some  targets  in  the  time  prescribed  (22-23  days). 
Certain  targets  are  most  efficiently  serviced  by  Weapon  1  or  Weapon  2, 
which  are  very  accurate;  at  low  levels  of  each  (the  front  portion  of  the 
box),  the  less-accurate  substitute — ^Weapon  3 — cannot  be  utilized 
effectively  against  these  targets  within  the  time  limit. 

What  about  the  region  in  the  back  of  the  box?  Here,  both  Weapon  1  and 
Weapon  2  are  at  relatively  high  levels,  so  why  aren't  there  any 
solutions?  The  reason  has  to  do  with  the  availability  of  platforms  to 
deliver  these  numbers  of  munitions  to  their  targets.  Simply  put,  in  this 
region  we  do  not  have  sufficient  aircraft  or  sorties  to  use  all  these 
weapons.  This  suggests  that,  were  we  to  impose  a  more  favorable  TPFD 
schedule  (in  which  more  platforms  arrived  earlier  in  the  campaign), 
enable  a  higher  sortie  rate,  or  change  the  platform  mix  in  theater,  we 
would  begin  to  find  solutions  in  the  back  region.  Here  is  a  connection 
between  simply  determining  the  weapon  mix  for  a  given  force  (similar 
to  DAWMS,  phase  one)  and  examining  alternative  forces  along  with  the 
weapon  mix  (phase  two). 

Another  kind  of  tradeoff  can  be  best  seen  by  looking  at  this  same  set  of 
tradeoffs  from  above,  as  we  do  in  the  next  slide. 
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This  chart  shows  the  tradeoff  between  Weapon  1  and  Weapon  2,  with 
Weapon  3  (the  third  dimension)  depicted  with  shading.  We  can  see  the 
original  solution,  as  well  as  the  features  described  in  the  last  slide — the 
skewness  in  Weapon  3  in  one  corner,  as  well  as  the  areas  where  the 
weapon  mix  is  not  feasible  (the  light  boxes). 

This  graph  gives  us  a  look  into  the  nature  of  the  tradeoff  between 
Weapons  1  and  2.  Notice  the  distinct  "dogleg"  in  the  set  of  potential 
weapon  mixes.  What  this  indicates  is  a  relatively  low  level  of  substituta¬ 
bility  between  these  weapons  in  the  more  extreme  regions.  For  instance, 
a  relatively  small  decrease  in  Weapon  2,  when  the  quantity  is  already 
low  (say  4,000),  will  require  a  disproportionately  large  increase  in  the 
number  of  Weapon  1  to  make  up  the  difference.  The  converse  is  true  at 
low  Weapon  1  and  high  Weapon  2  levels. 

So  how  "good"  was  the  original  solution?  Given  this  picture  of  the  full 
set  of  possible  weapon  mixes,  the  solution  found  using  traditional 
analysis  is  potentially  not  a  preferred  one.  For  example,  we  may  reduce 
the  sheer  quantity  of  weapons  needed  by  taking  a  large  decrease  in 
Weapon  1  and  only  a  small  increase  in  Weapon  2.  Thus,  this  chart  tells 
us  something  about  the  relative  efficiency  of  different  solutions. 

None  of  the  preceding  information — about  the  whole  set  of  solutions, 
the  regions  of  feasibility  and  infeasibility,  the  nature  of  the  weapon 
tradeoffs,  and  the  relative  efficiency  of  alternative  solutions — can  be 
derived  from  the  traditional  analysis,  which  provides  the  single, 
optimum  solution. 
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Before  turning  to  a  more  general  discussion  of  the  type  of  analysis  we 
have  performed  here,  we  return  to  the  troubling  result  we  observed  as 
Weapon  1  reliability  was  reduced  and  the  Weapon  1  component  of  the 
mix  increased  and  decreased  repeatedly. 
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In  the  preceding  two  charts,  we  have  looked  at  the  set  of  weapon  mixes 
that  achieve  a  fixed  objective  value  and  a  fixed  Weapon  1  reliability, 
sortie  rate,  and  deployment  schedule  (all  at  their  nominal  values). 

What  happens  as  we  degrade  one  of  these,  say.  Weapon  1  weapon 
reliability? 

First,  we  now  know  that  we  should  look  at  a  whole  set  of  solutions  for 
each  Weapon  1  reliability  value;  the  point-to-point  comparison  from 
our  original  sensitivity  analysis  was  incomplete.  The  previous  charts 
showed  the  trade  space  for  100  percent  reliability.  What  is  the  trade 
space  for  lower  Weapon  1  reliability?  Should  we  still  worry  about  the 
troubling,  nonmonotonic  results  that  we  originally  observed  as  we 
change  Weapon  1  reliability? 

To  address  the  first  question,  the  chart  superimposes  the  set  of  weapon 
mix  solutions  for  the  baseline  (100  percent)  case  on  the  cases  in  which 
Weapon  1  reliability  is  90  percent  and  85  percent  (sorties  and 
deployment  are  maintained  constant  at  100  percent  in  all  cases),  all  of 
which  correspond  to  an  objective  value  of  22-23  days  to  complete  the 
campaign. 

For  visual  ease,  we  look  at  only  two  dimensions  (Weapons  1  and  3). 

The  regions  are  inclusive;  that  is,  all  points  within  the  darkest  region's 
perimeter  are  solutions  for  Weapon  1  reliability  of  100  percent,  all 
points  within  the  next  lightest  region's  perimeter  are  solutions  at 
90  percent,  and  so  on.  In  addition,  the  site  of  the  solutions  calculated 
by  the  traditional  use  of  the  model  for  each  of  these  parameter  values  is 
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indicated.  Note  the  counterintuitive  change  among  these  "optima" 
from  using  more,  to  fewer,  to  more  of  Weapon  1,  as  seen  earlier. 

This  graph  explains  the  troubling  results  we  saw  before.  It  is  clear  that 
many  possible  solutions  to  our  problem  yield  the  same  objective  value 
across  these  values  of  Weapon  1  reliability.  Many  of  them  are 
coincident  (those  near-optimal  mixes  in  the  lightest  region  are  also 
near-optimal  solutions  for  the  90  percent  and  100  percent  case).  When 
we  ran  the  model  in  the  traditional  way,  however,  this  was  not 
indicated.  The  model  (arbitrarily)  calculated  only  one  of  the  many 
possible  weapon  mixes  in  each  case,  and  the  resulting  three  mixes 
happened  to  first  require  more  of  Weapon  1,  then  less,  then  more  as 
reliability  dropped  off.  Because  traditional  sensitivity  analysis 
provided  limited  results,  we  incorrectly  interpreted  the  non¬ 
monotonicity  of  the  results  as  being  troubling,  and,  perhaps,  as  even 
indicative  of  errors  in  the  model  or  data. 

These  more  comprehensive  results  explain  what  is  really  happening. 
The  reduction  in  Weapon  1  reliability  is  actually  changing  the  set  of 
weapon  mixes  that  can  accomplish  the  campaign  objectives. 
Furthermore,  we  are  also  provided  with  the  answer  to  what  we  should 
do  at  fairly  modest  amounts  of  uncertainty  in  Weapon  1  reliability:  Just 
pick  a  point  somewhere  in  the  middle  of  the  lightest  region.  We  know 
that  this  mix  is  an  optimal  solution  across  all  three  values  of  the 
parameter.  In  other  words,  we  don't  need  to  agonize  over  the  accuracy 
or  precision  in  Weapon  1  reliability  (which  is  highly  uncertain  in  future 
campaigns).  Instead,  we  can  select  a  weapon  mix  that  is  robust  across  a 
range  of  Weapon  1  reliability  conditions.  In  other  words,  we  can  use 
our  decision  to  select  a  weapon  mix  to  not  only  fulfill  campaign 
objectives,  but  also  to  resolve  uncertainties  in  the  scenario  conditions. 

This  example  sums  up  the  intrinsic  difference  between  traditional 
analysis  and  our  nontraditional  approach.  Traditional  analysis  found  a 
single  point  (somewhat  arbitrarily,  as  it  turns  out),  which  we  regarded 
as  the  solution  and  examined  the  variation  around  it;  with  our 
approach,  we  systematically  search  for  all  the  possible  optimal  and 
near-optimal  solutions,  then  select  a  preferred  one.  The  traditional 
approach  works  from  the  inside  out.  We  are  suggesting  that  working 
from  the  outside — the  big  picture — and  selecting  a  solution  within 
provides  important  additional  information  for  both  analysts  and 
decisionmakers.  An  additional  significant  difference  with  this 
approach  is  the  ability  to  choose  the  most  advantageous  weapon  mix  from 
the  set  of  good  solutions  based  on  a  variety  of  preferences  (e.g.,  costs, 
risk,  operational  considerations)  that  may  be  difficult  to  capture  in  a 
single  model. 
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We  call  the  methodology  we  have  just  employed  "exploratory 
analysis."  In  this  next  section  we  will  explain  the  generality  of  the 
approach  and  the  details  of  how  it  can  be  applied. 
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Exploratory  Analysis 


Searching  for  robust  solutions  across  a  variety  of 
plausible  parameter  values,  scenarios,  and 
representations. 

•  Changes  how  models  are  used 

•  Requires  large-scale  computational  experiments 
•  Advantages 

•  Increases  decisionmaking  flexibility 

•  Assesses  robustness  across  contingencies 

•  Neutralizes  risk 

-  imperfect  information 

-  unpredictable  future 


Exploratory  analysis  searches  across  the  space  of  scenarios,  decisions, 
and  measures  of  effectiveness  in  search  of  robustness.  It  allows  us  to 
look  at  robustness  in  a  variety  of  ways.  So  far  in  our  examples,  we  have 
determined  weapon  mix  (decision  recommendations)  solutions  that  are 
robust  across  the  variability  or  uncertainty  in  Weapon  1  reliability  (an 
input  value).  We  might  also  look  for  decision  recommendations  that 
are  common  to  different  scenarios — for  instance,  across  two  or  more 
theaters  of  operations.  We  could  consider  looking  for  decision 
recommendations  that  are  robust  across  models.  For  example,  suppose 
we  are  not  sure  which  of  three  different  models'  solutions  are  "best" 
(however  that  might  be  defined).  By  sampling  across  the  decision  space 
produced  by  all  three  and  choosing  a  solution  common  to  all,  we  no 
longer  have  to  argue  the  highly  subjective  question  of  whose  model  is 
best. 

How  is  exploratory  modeling  different  from  traditional  analysis?  First, 
it  changes  the  way  the  model  is  used.  The  model  is  run  many  times 
with  many  different  input  levels,  which  requires  fairly  large  numbers  of 
computational  experiments  (perhaps  tens  or  hundreds  of  thousands). 
These  experiments  are  made  possible  in  our  example  by  today's 
computing  environment — high-powered,  networked  workstations. 

Second,  as  alluded  to  before,  exploratory  modeling  represents  a  funda¬ 
mentally  different  way  of  looking  at  the  problem.  While  traditional 
analysis  works  in  a  sense  from  the  inside  out — the  solution  is  found  and 
the  area  around  it  is  examined — exploratory  modeling  allows  an 
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"outside  in"  approach:  A  large  space  in  the  domain  of  interest  (and  all 
the  solutions  in  it)  is  examined,  then  a  solution  is  selected.  The  next 
few  pages  address  this  difference  in  more  detail. 

This  more  inductive  method  to  solving  the  problem  has  a  number  of 
advantages  over  traditional  analysis.  Using  the  weapon  mix  problem, 
we  will  demonstrate  in  the  last  section  of  this  documented  briefing  that 
the  advantages  include  greater  decisionmaking  flexibility,  ability  to 
assess  robustness,  and  risk  neutralization. 
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How  Does  Exploratory  Analysis  Differ  I 

from  Sensitivity  Analysis?  (1)  { 

Sensitivitv  Analysis 

Exploratory  Analysis 

Scenarios  and  .  . 
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Conditions  Variables 
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This  diagram  provides  one  view  of  how  exploratory  analysis  differs 
from  traditional  sensitivity  analysis.  Simply  put,  sensitivity  analysis 
tries  to  measure  the  impact  of  changing  inputs  on  outputs.  More 
carefully  and  broadly,  sensitivity  analysis  tries  to  capture  how  measures 
of  effectiveness  change  as  either  the  scenario  or  the  decisions  change.® 

Exploratory  analysis  takes  a  broader  perspective  and  not  only  considers 
much  more  comprehensive  sensitivity  analysis,  but  a  greater  set  of 
effects.  We  can  determine  the  impact  of  a  particular  measure  (such  as 
days  to  complete  a  campaign)  on  the  inputs  or  the  decision  variables 
(such  as  the  mix  of  weapons).  This  effect  (how  changes  to  the  output 
affect  inputs)  could  be  thought  of  as  sensitivity  analysis  in  reverse.  But 
this  is  not  all.  We  may  also  desire  to  capture  the  interaction  between  or 
within  the  set  of  values  that  make  up  the  scenario,  the  decision  variables, 
or  the  measures  of  effectiveness. 


®  We  recognize  that  in  a  prescriptive  model  (such  as  an  optimizing  model)  the  decision 
variables  are  outputs  whereas  in  a  descriptive  model  (such  as  a  simulation)  the 
decision  variables  are  part  of  the  inputs.  For  sensitivity  analysis,  we  typically  desire  to 
know  what  impact  the  decision  variables  have  on  measures  of  performance,  regardless 
of  whether  they  are  inputs  or  outputs. 
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On  a  slightly  more  mathematical  level,  we  see  that  exploratory  analysis 
can  be  distinguished  from  sensitivity  analysis  in  terms  of  the  implied 
directions  in  which  we  can  capture  changes  in  values.  Say  that  the 
model  we  are  using  can  be  represented  by  the  function/(X).  While 
traditional  sensitivity  analysis  and  exploratory  analysis  concern 
themselves  with  changing  the  inputs  X  and  seeing  the  effect  on  the 
output(s)  Y  via/(X),  exploratory  analysis  goes  further  and  allows  us  to 
consider  the  range  of  X  (the  combinations  of  inputs)  that  corresponds  to 
a  particular  range  of  Y  via/\Y).  We  can  also  look  at  interactions 
among  the  set  of  inputs  X  and  the  set  of  outputs  Y.  Note  that 
exploratory  analysis  goes  beyond  sensitivity  analysis  to  the  extent  that 
it  allows  us  to  examine  the  comparative  statics  of  our  problem  as 
opposed  to  just  the  partial  derivatives  of  the  objective  function. 
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Outline 


•  A  traditional  analysis  with  some 
conclusions 

•  What  is  really  happening:  an 
introduction  to  exploratory  analysis 

•  The  methodology  of  exploratory 
analysis 

How  can  exploratory  analysis  help 
decisionmakers? 

•  Uses  and  requirements  of  exploratory 
analysis 
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This  section  gives  a  few  concrete  examples  of  how  exploratory  analysis 
can  help  the  decisionmaker. 
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This  chart  represents  in  numerical  form  the  data  presented  in  prior 
graphs.  Levels  of  Weapon  1  are  on  the  horizontal  axis;  levels  of 
Weapon  2  are  on  the  vertical  axis;  and  the  numbers  in  the  inside  cells 
correspond  to  Weapon  3  (all  weapons  are  rounded  up  to  the  nearest 
500).  Blank  cells  are  the  infeasible  combinations.  Under  traditional 
analysis,  we  would  have  gotten  the  one  solution  pointed  to  with  the 
arrow. 

How  are  additional  weapon  mixes  found  via  exploratory  analysis 
superior  to  the  single-point  solution  found  by  conventional  means? 
Consider  first  an  operational  commander.  While  a  commander  does 
not  buy  weapons,  his  staff  does  perform  analysis  to  determine  needs. 
Suppose  the  commander's  staff's  traditional  analysis  has  come  up  with 
the  single  point,  and  that  for  whatever  reason  (for  example,  jamming  or 
technological  risk),  the  commander  is  uncomfortable  with  using  so 
many  of  Weapon  1  and  would  prefer  to  substitute  Weapon  2  for  them, 
if  possible.  With  the  single-point  solution,  the  commander  has  no  idea 
if  by  using  more  Weapon  2  there  are  in  fact  combinations  of  Weapons  1 
and  3  that  allow  him  to  complete  his  mission  in  approximately  the 
same  amount  of  time.  If,  however,  the  commander's  staff  had  used 
exploratory  analysis  to  generate  the  table  above,  the  commander  could 
readily  choose  among  a  large  number  of  solutions  in  the  upper  right- 
hand  comer  that  require  relatively  few  of  Weapon  1. 
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Additionally,  from  an  operational  perspective,  this  table  of  tradeoffs 
would  also  prove  useful  in  the  event  that  in-theater  inventories  of  a 
particular  munition  were  below  the  single-point,  "optimal"  level. 
Could  other  munitions  be  used  to  substitute  for  this  deficiency?  Will 
the  munition  deficiency  prohibit  the  commander  from  accomplishing 
mission  objectives?  A  table  like  this  readily  provides  the  alternatives 
that  permit  the  same  level  of  mission  success. 
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Decision  Space:  Campaign  Finishes  in  22-23  Days 
with  Hypothetical  Budget  Constraint 
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Exploratory  analysis  can  also  increase  the  force  programmer's 
flexibility.  This  chart  exhibits  the  same  solutions  as  on  the  last,  but 
shades  about  half  of  them.  The  shaded  cells  represent  solutions  that  fall 
within  a  notional  budget  constraint  (the  less  costly  weapon  mixes).  If 
this  were  the  actual  budget  constraint,  flexibility  would  be  enhanced  by 
allowing  selection  from  the  variety  of  mixes  (including  the  least 
expensive  one)  feasible  for  the  budget,  as  opposed  to  just  hoping  that 
the  one  solution  found  with  traditional  analysis  might  fit. 

Each  of  these  examples  demonstrates  how  exploratory  analysis  allows 
additional  preferences  to  be  included  and  their  impact  displayed,  thus 
improving  decisionmaking  flexibility. 
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Robustness  Across  Contingencies 


Weapon  1 

I  Solution  for  SWA  Q  Solution  for  NEA  (hypothetical) 

□  Common  Solution 
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Exploring  the  space  of  solutions  provides  the  ability  to  assess 
robustness  better  across  two  contingencies  and  to  allow  the 
decisionmaker  to  choose  a  weapon  mix  that  is  robust  for  either  case. 
This  chart  considers  two  distinct  contingencies:  SWA  and  NEA.^ 

If  we  are  uncertain  as  to  which  might  occur,  how  do  we  select  the 
weapon  mix  to  have  in  inventory?  Buying  enough  for  each  one 
separately  leads  to  an  over  costly  solution.  Traditional  analysis  can 
determine  the  requisite  inventory  for  each,  without  buying  overlapping 
types  or  quantities.  Although  better,  the  traditional  approach  may  also 
lead  to  excessive  inventory. 

Using  exploratory  analysis,  we  can  display  the  full  trade  space  for  each 
contingency,  as  shown  above.  The  darker  cells  are  the  solutions  we 
found  for  SWA.  The  checkered  cells  correspond  to  notional  solutions 
for  another  theater,  NEA.  Clearly,  given  these  two  solution  sets,  we  can 
select  a  weapon  mix  that  provides  success  in  either  SWA  or  NEA  (the 
medium-shaded  cells);  we  can  see  the  robustness  of  weapon  mixes 
across  contingencies. 

It  is  possible,  of  course,  that  the  solution  set  for  the  second  theater  is 
disjoint  from  the  first,  so  no  common  weapon  mixes  exist.  In  such  a 
case,  exploring  the  space  of  solutions  is  still  extremely  valuable.  With 


^The  definition  of  the  two  contingencies  in  this  example  are  completely  arbitrary. 
One  could  have  been  a  combined  SWA  and  NEA,  the  second  an  alternative  major 
regional  contingency  (MRC)  and  a  mission  other  than  war. 
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traditional  analysis,  we  calculate  only  single-point  solutions  for  each 
theater,  which  are  arbitrarily  far  apart.  With  exploratory  analysis,  the 
full  set  of  possibilities  is  seen,  so  that  a  weapon  mix  closest  to  both  can 
be  selected.  Implications  of  the  risk  are  more  clearly  understood. 
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An  example  of  how  exploratory  analysis  can  neutralize  risk  has  already 
been  given,  but  we  do  so  more  explicitly  here.  This  chart  contains  just  a 
few  cells  from  the  weapon  mix  table  shown  previously.  These  weapon 
mixes  all  correspond  to  the  nominal  100  percent  Weapon  1  reliability. 
But  what  if  Weapon  1  reliability  is  95  percent  or,  perhaps,  90  percent? 
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Weapon  1  Reliability  =  90% 
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This  chart  shows  the  weapon  mixes  that  are  solutions  for  the  90  percent 
Weapon  1  reliability  case.  The  neutralization  of  the  risk  or  uncertainty 
about  Weapon  1  can  be  accomplished  by  selecting  a  solution  that  is  the 
same  in  this  and  the  previous  chart.  The  next  chart  compares  the 
weapon  mixes  above  to  those  at  100  percent  reliability  to  find  a  few  of 
these  common  mixes. 
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The  common  solutions  are  the  darker  cells:  Any  of  these  weapon  mixes 
assures  us  of  achieving  the  same  outcome  of  22-23  days  to  complete  the 
campaign  regardless  of  whether  Weapon  1  is  completely  or  somewhat 
less  effective.  It  might  be  noted  that  the  common  cells  here  have  a 
small  variance  in  the  numbers  of  Weapon  3.  For  ease  of  exposition,  the 
solutions  are  made  common  by  rounding  up  to  the  higher  Weapon  3 
level  of  the  two.  Given  more  exhaustive  sampling  for  solutions,  we  are 
likely  to  find  weapon  mixes  for  all  three  weapon  types  that  are  truly 
common  across  levels  of  Weapon  1  reliability. 
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Outline 


•  A  traditional  analysis  with  some 
conclusions 

•  What  is  really  happening:  an 
introduction  to  exploratory  analysis 

•  The  methodology  of  exploratory 
analysis 

•  How  can  exploratory  analysis  help 
decisionmakers? 

•  Uses  and  requirements  of  exploratory 
analysis 
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The  final  section  discusses  uses  and  requirements  of  exploratory 
analysis. 
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•  Comprehensive  analysis  of  complex  or  poorly 
understood  systems 

-  Military 

Force  structure,  logistics,  adaptive  planning 

-  Nonmilitary 

Healthcare,  education,  investment  strategy 

•  Model  validation 

-  Validation  of  simple  deductive  models  containing  the 
inputs  of  interest 

-  Validation  of  the  underlying  simulation  or  optimizer 
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In  this  briefing  we  have  used  exploratory  analysis  only  in  a  limited  way. 
The  full  extent  of  this  approach  goes  beyond  our  examples  both  in  scale 
and  in  scope.  This  methodology  can  and  has  been  used  to  address  a 
wide  variety  of  problems.  For  example,  RAND  projects  have  used 
exploratory  analysis  in  analyses  of  military  strategy,  force  structure, 
logistics,  and  adaptive  defense  planning.®  In  addition,  it  has  been 
applied  to  such  diverse  nonmilitary  topics  as  science  and  technology 
investment  strategy,  drug  control  policy,  global  warming,®  and  the  future 
of  higher  education. 

We  note  again  that  beyond  providing  more  effective  uses  of  models  built 
for  analysis,  exploratory  analysis  also  provides  an  effective  vehicle  for 
improvement  of  the  models  themselves.  After  all,  the  best  way  to 
uncover  anomalous  model  behavior  is  via  large-scale  systematic 
experimentation  under  a  wide  variety  of  conditions  and  input  values. 
Exploratory  analysis  can  be  used  as  an  effective  tool  in  the  verification 
and  validation  process. 


®  See  James  A.  Dewar,  Steven  C.  Bankes,  James  S.  Hodges,  Thomas  Lucas, 

Desmond  K.  Saunders-Newton,  and  Patrick  Vye,  Credible  Uses  of  the  Distributed 
Interactive  Simulation  (DIS)  Environment,  Santa  Monica,  Calif.:  RAND,  MR-607-A,  1996, 
or  Steve  Bankes,  Modeling  and  Simulation  Infrastructure  to  Support  Adaptive  Planning  in  the 
Air  Force,  Santa  Monica,  Calif.:  RAND,  DB-210-AF,  1997. 

®  See  Robert  J.  Lempert,  Michael  E.  Schlesinger,  and  Steve  Bankes,  "When  We  Don't 
Know  the  Costs  or  the  Benefits:  Adaptive  Strategies  for  Abating  Climate  Change," 
Climatic  Change,  Vol.  33, 1996,  pp.  235-274. 
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What  Does  Exploratory  Analysis  Require?  \ 


•  Model  run  time:  35-150  minutes  on  SPARC  20 

•  Number  of  runs:  1,150 

•  Number  of  work  stations:  8-13 

•  Totai  ciock:  4-5  days 

•  Analyst  time:  10  weeks 
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We  have  mentioned  throughout  that  exploratory  analysis  requires 
large-scale  computational  experiments.  To  make  this  more  concrete, 
this  chart  summarizes  what  exactly  went  into  the  explorations 
performed  over  the  course  of  the  analysis  described  in  this  briefing. 
Given  the  model  rim  time,  number  of  work  stations,  and  scope  of  this 
project,  the  number  of  runs,  while  substantial,  was  not  prohibitive.  In 
other  analyses  and  with  other  models,  the  time  required  to  perform 
comprehensive  exploratory  analysis  may  or  may  not  be  affordable. 

In  a  typical  analysis,  it  is  not  feasible  to  significantly  alter  run  time; 
additionally,  time  for  completion  of  the  analysis  is  often  fixed.  Given 
this  information  and  the  number  of  work  stations  available,  the  maxi¬ 
mum  number  of  model  runs  can  be  determined.  We  seek  to  set  up  the 
maximum  number  of  runs  so  that  as  much  useful  information  as 
possible  is  provided  for  the  decisionmaking  process. 

The  run  time  in  our  example  could  have  been  better  spent  if  we  had 
used  more  clever  sampling  and  searching  techniques.  Referring  back  to 
the  weapon  mix  trade  spaces  shown  on  pages  25  and  27,  we  note  that 
the  critical  information  on  these  charts  is  the  boundary  and  the  area 
inside  the  boundary.  With  our  limited  set  of  tools  to  support 
exploratory  analysis,  we  were  required  to  calculate  not  only  the  cells 
where  there  was  an  alternative  weapon  mix,  but  also  the  cells  where 
there  were  none  to  find.  This  brute  force  approach  was  necessary  but 
inefficient.  Given  a  better  infrastructure  to  perform  exploratory 
analysis,  either  the  same  information  could  have  been 
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obtained  from  far  fewer  runs,  or  the  same  number  of  runs  could  have 
been  used  to  produce  even  more  information. 
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Infrastructure  Development  Is  Needed  for 
Broader  Application  of  Exploratory  Analysis 

•  Large-scale  computational  experiments  are  limited  by 

-  Conceptual  definition  of  the  experiments 

-  Generating  and  managing  experiments  ■; 

-  Handling  and  analyzing  large  quantities  of  output  data 

•  Infrastructural  improvements  that  address  these  limits 
would  be  in  the  following  areas 

-  Operating  system  interfacing  for  multiple  machine  processing  of  cases 

-  Automating  case  generation  and  model  specification 

-  Database  management 

-  Visualization  tools  for  multidimensional  results 

-  Search  and  sampling  methods 

- -  43  rand 

‘ .  .  . . ~ . z)  ^  ^  . 

The  case  study  presented  in  this  documented  briefing,  while  effective  in 
demonstrating  some  of  the  advantages  of  exploratory  analysis,  was  not 
a  taxing  exercise  of  the  methodology.  It  is  not  difficult  to  imagine 
studies  in  which  a  far  greater  variety  of  uncertainties,  decision  options, 
or  measures  of  effectiveness  must  be  considered.  For  such  studies,  it 
might  be  difficult  to  define  precisely  what  is  the  space  to  explore  vis-a- 
vis  the  computational  experiments.  Other  problems  can  arise  in 
manipulating  large  volumes  of  data  going  into  the  experiments  as  well 
as  the  consumption  and  interpretation  of  potentially  huge  quantities  of 
output  generated. 

In  our  case  we  observed  the  interconnected  effects  of  changes  in  just 
three  scenario  parameters  (weapon  reliability,  sortie  rates,  and 
deployment  rates),  three  decision  options  (the  numbers  of  three 
different  weapon  types),  and  one  output  measure  (days  to  complete  the 
campaign).^®  Certainly,  these  are  not  the  only  interactions  we  could 
have  looked  at;  we  limited  ourselves  in  this  way  precisely  because  it 
kept  us  within  a  manageable  level  of  dimensionality  (of  the  trade  space) 
to  conduct  an  example  analysis  given  the  present  state  of  the 
methodology's  infrastructure. 

What  practical  steps  could  be  taken  to  improve  our  ability  to  tackle 
more  highly  dimensional  problems  with  exploratory  analysis?  Several 


also  included  measures  of  cost  Etnd  risk  in  the  analysis,  but  this  was  done 
externally  to  the  principal  exploration. 
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are  immediately  clear.  First,  we  could  provide  operation  system 
interface  for  multiple-machine  processing  of  model  cases.  Exploratory 
analysis  relies  on  large  computational  experiments  that  can  be  most 
efficiently  conducted  across  a  large  number  of  networked  computers. 
This  step  would  provide  a  standard  means  for  accessing  and 
controlling  UNIX-based  machines  on  the  network  by  using  a 
generalized,  transportable  interface  to  the  operation  system.  This 
controller  would  provide  coordinated  access  to  all  the  workstations  on 
a  network  whether  there  were  tens  or  hundreds  available. 

Second,  we  could  devise  and  implement  software  to  facilitate  case 
generation  and  model  specification.  This  would  provide  automation  for 
the  generation  of  the  computational  experiments. 

Third,  we  could  adopt  and  implement  software  to  support  result 
databases.  Exploratory  analyses  generate  large  quantities  of  data  that 
must  be  stored  and  manipulated.  These  software  tools  would  provide 
standardized  facilities  for  handling  these  databases,  including 
improved  capabilities  for  output  database  maintenance  and  the  ability 
to  use  multiple  hardware  and  database  systems. 

Fourth,  we  could  find  or  develop  improved  visualization  tools  for 
multidimensional  model  results.  As  noted  above,  exploratory  analysis 
calculates  the  effect  of  interactions  among  many  important  problem 
parameters  such  as  force  levels,  system  effectiveness,  timing,  and 
tactics.  Being  able  to  understand  and  communicate  these 
multidimensional  effects  can  be  difficult;  this  step  would  ideally 
provide  a  portfolio  of  visualization  tools  that  were  readily  available  as 
part  of  the  basic  exploratory  analytic  environment. 

Fifth,  we  could  implement  a  portfolio  of  search  and  sampling  methods. 
The  exploratory  analysis  conducted  here  used  brute  force  methods  for 
discovering  the  critical  points  at  which  decisions  change.  This  task 
would  research  algorithmic  options  for  exploring  high-dimensional 
model  spaces  automatically  and  finding  these  critical  points  much  more 
efficiently.  As  a  result  of  these  tools,  study  results  could  be  derived 
more  rapidly  and  more  extensive  analysis  could  be  performed. 

After  these  steps  were  taken,  we  would  then  likely  turn  our  attention  to 
methods  for  transferring  the  tools  of  exploratory  analysis  to  those 
outside  our  immediate  analytic  environment.  In  this  way,  the 
advantages  of  this  methodology  could  be  exploited  widely. 
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Conclusions 


•  Traditional  analysis  provides  useful  but  limited 
information  for  a  decisionmaker. 

•  Improving  the  model  or  data  alone  can  be 
inefficient  and  may  not  solve  the  problem. 

•  Advantages  of  exploratory  analysis 

-  Decisionmaking  flexibility 

-  Robustness  across  contingencies 

-  Risk  neutralization 

-  Greater  understanding  of  the  model 
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The  conclusions  described  at  the  outset  can  now  be  put  into  better 
focus.  First,  we  have  seen  that  exploratory  analysis  is  not  at  odds  with 
traditional  analysis,  but,  rather,  that  it  expands  traditional  analysis  to 
provide  greater  information  to  the  decisionmaker.  Essentially,  what  we 
have  shown  is  that,  to  assemble  an  adequate  body  of  information  about 
the  weapon  mix  problem,  we  need  to  perform  traditional  analysis  many 
times  in  a  schematic  way  across  the  entire  space  of  potential  solutions. 
Second,  we  have  demonstrated  how  exploratory  analysis  can  deal  with 
problems,  such  as  troubling  results  commonly  encountered  in 
traditional  analysis.  Our  demonstration  has  also  pointed  out  that  the 
responses  often  made  to  such  problems  (absent  exploration)  are  not 
likely  to  be  effective,  since  the  true  nature  of  the  problem  is  not 
inadequacies  of  the  model  or  data,  but  inadequacies  of  the  analysis 
(how  the  model  has  been  used).  Finally,  we  have  demonstrated  that 
exploratory  analysis  provides  some  useful  advantages,  including 
allowing  greater  flexibility  and  robustness,  lowering  risk,  and 
improving  understanding. 

The  central  concept  behind  exploratory  analysis  is  relatively  simple. 

For  the  test  case  we  have  described  here,  the  benefits  are  easy  to  see, 
inexpensive  to  obtain,  yet  potentially  quite  costly  to  neglect. 
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